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ABSTRACT 

This document presents the third year evaluation of 
the CAI/Cooperative Learning Project. The purpose of this project is 
to develop a national model for integrating computer-assisted 
instruction (CAi) through an integrated learning system. The project 
is a collaborative effort by two Pennsylvania school districts, 
Hatboro-Horsham and Pittston Area, and Research for Better Schools 
(RBS) . This report begins wit an introduction that provides 
background information on the project and the evaluation questions 
addressed by the study. Next, the evaluation design and procedures 
are described including design, student sample, instrumentation, and. 
data collection and analysis. The findings of the evaluation 
questions for year three are presented in two sub-sections: program 
implementation and program outcomes. Finally, general conclusions 
which relate to the project's evaluation questions as well as 
implications of the evaluation's findings and conclusions are 
presented. Included in the appendix are the surveys used in the 
evaluation as well as the mean survey responses for rated items. 
(JLB) 
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Introduction 

This report represents a third-year evaluation of the "CAI/Cooperative 
Learning Project." The three-year project is a collaborative effort by two 
Pennsylvania school districts, the Pittston Area School District and the 
Hatboro-Horsham School District, and Research for Bstter Schools (RBS) . It is 
federally supported by an Innovation in Education Program Grant. 

The introductory section of the report provides some background informa- 
tion on the project and the evaluation questions addressed by the study; later 
sections of the report describe the evaluation design and procedures P 
findings, and some conclusions and implications based on these findings. 
Background 

Both the Pittsteti and Hatboro-Horsham School Districts had previously 
been involved in successful efforts to implement computer-assisted instruc- 
tional technology. In order to further adopt this technology, in combination 
with innovative educational practices, the two districts jointly proposed this 
collaborative project to the Fund for Innovation in Education. Specifically, 
the project proposed to integrate advanced integrated learning system (ILS) 
technology with cooperative teaching in the classroom and collaborative 
learning in the home. It was anticipated that the project could potentially 
serve as a model for effective computer-assisted instruction that could be 
nationally validated, disseminated, and adopted by school districts across the 
nation. To realize this potential, the project needed a sound evaluation plan 
capable of demonstrating the full extent of its effectiveness. Because of its 
history in evaluating computer-assisted instructional programs, RBS was 
invited to collaborate with the two districts in the project, serving as a 
third-party evaluator. 
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The project was to include the acquisition of computer hardware and soft- 
ware, initial and on-going teacher training, achievement and attitudinal data 
collection and analysis, and dissemination. For their software, the two 
school districts selected a computer-based learning system marketed by Jostens 
Learning Corporation. 
Evaluation Questions 

The purpose of the CAI/Cooperative Learning Project is to develop a 
replicable, independently validated, national model for integrating computer- 
assisted instruction through an integrated learning system that includes 
inquiry-based, hyper-media learning and cooperative learning techniques. An 
original feature of the project is the inclusion of cooperative education, 
creating a collaborative learning environment both within the classroom and at 
home. Based on the project goals, as specified in the project proposal, three 
outcome evaluation questions were formed to focus the evaluation study. They 
were : 

1. To what extent does the project enhance mathematics and language arts 
achievement for the participating students as measured by an increase 
in standardized test scores? 

2. To what extent does the project enhance positive student and parent 
attitudes toward learning, the integrated learning system, and educa- 
tion in general? 

3. To what extent does the project enhance positive teacher and adminis- 
trator attitudes toward the. integrated learning system and coopera- 
tive learning? 

The project's first and second year evaluations were primarily descrip- 
tive (Beyer, 1991, 1992). As a result of delayed funding and difficulties in 
ordering and receiving the necessary hardware and software, the project was 
not initiated until the second half of its first year and, during this time, 
was not fully operational. An implementation focus for the first year evalua- 
tion was designed to help ensure that prerequisite conditions for proper 
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program implementation were met. During year two, the two participating 
school districts experienced many difficulties with the Jostens Learning 
Corporations 9 computer-based learning system. Because of these unexpected 
problems, which involved both the computer software and hardware, the project 
reached only a minimal level of operation; the evaluation of the project's 
second year had an implementation focus as well. During year three, the 
unexpected and frustrating difficulties with the computer-based learning 
system were less prevalent and the project was more fully implemented in the 
two middle schools. However, because this is a summary report of a three-year 
project, and critical implementation issues remained a key concern, the 
process evaluation question included in the first and second year reports is 
also included in this third year report. Thus, a fourth evaluation question 
was added: 

4. To what extent was the project implemented as planned? 
A discussion of the findings related to this evaluation question will be 
presented first to set the stage for interpreting the outcome evaluation 
findings . 

Evaluation Design and Procedures 
This section describes the evaluation design, student sample, instrumen- 
tation, and data collection and analysis undertaken by RBS in connection with 
its third year evaluation of the CAI /Cooperative Learning Project implemented 
in two middle schools, one in the Pittston Area School District and one in the 
Hatboro-Horsham School District. 
Design 

The approach used to address the evaluation questions was to include 
quantitative data collection of curriculum-embedded tests and standardized 
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achievement tests. In addition, implementation and attitude measures were to 



be obtained through survey data collected at the end of the school year and 
through observations and informal interviews made during periodic visits to 
each of the sites. A pretest-posttest control group design was to be used to 
enhance the validity of the findings, although there were differences in the 
nature of the control or comparison group at each school. For the Hatboro- 
Horsham School District, the control group was to be constituted from students 
within the same school who had no contact with computer-assisted instruction. 
For the Pittston Area School District, the control group was to consist of 
students within the school who had minimal contact with computer-assisted 
instruction. 

A number of changes were made in the evaluation design originally pro- 
posed. Major changes and their rationales are listed below. In terms of the 
proposed evaluation plan for collecting outcome data: 

• The TELLS was not an outcome measure, as planned, as the state 
(Pennsylvania) discontinued this testing program. However, other 
standardized test data and curriculunv s embedded test data were 
collected and analyzed as planned. 

• Standardized test data from the Hatboro-Horsham School District were 
not collected and analyzed for year two as this district experienced 
extensive implementation difficulties (see Beyer, 1992). 

• Parent attitude data from the Pittston Area School District were not 
included in the third year report as the district did not administer 
the parent surveys. 

• During year three, the Pittston Area School District developed reading 
and mathematics laboratories for implementing the project and sched- 
uled all students to participate, thus eliminating the control group 
from RBS' evaluation design. This was in response to teacher and 
teacher union concerns about students missing regularly scheduled 
classroom instructional time while working on the computer. 

• For the year three evaluation, in order to obtain richer and more 
meaningful data on teacher and administrator attitudes, interviews and 
observations were emphasized rather than attitude surveys (responses 
to open-ended survey questions were limited). 



In terms of the proposed evaluation plan for collecting process data: 

• The cooperative learning component was not fully addressed by the 
districts. Although during year three the Hatboro-Horsham School 
District did attempt to implement this component, adequate data were 
not available for analysis. 

• Time-on-t *sk data were not collected and analyzed as informal observa- 
tions indicated consistently high student engagement rates and very 
little variability. 

Further discussion of some of these issues can be found in the text. Also, 
because project implementation at each site was very different during year 
three as compared to years one and two, the focus of this evaluation report is 
on the project's third and final year. 
Student Sample 

Different strategies were, used at each site to select program and control 
groups. In the Hatboro-Horsham middle school each grade is divided into two 
heterogeneous teams, a red team and a black team. For year one, the program 
group was selected from only the red team and consisted of the 25 lowest- 
achieving students from each of the sixth, seventh, and eighth grades. All of 
these students had standardized achievement scores below grade level in mathe- 
matics and/or reading. The control group consisted of 25 comparable students 
from each of the black team's three grades (these students did not have compu- 
ters in their classes). For years two and three, the seventh and eighth grade 
program and control groups remained intact, to the extent possible; the new 
sixth grade groups were assigned using the year one strategy. 

For year one, the Pittston middle school was divided into two groups 
based on standardized achievement test scores, an at-risk or program group 
(i.e., scoring below the grade level median) and a not at-risk group. All 
students in the at-risk group were then randomly assigned to one of two 
groups, a "low use" group which was to receive 15 minutes of computer time per 
week, or a "high use" group which wa9 to receive 60 minutes of computer time 



per week (in 15 minute segments). There were approximately 50 students in 
each group at each grade level. To balance the low and high use groups, 
students assigned to one group for mathematics were to be assigned to the 
other group for reading. It should be noted that although the not at-risk 
group also spent time on the Jostens computer learning system, their data are 
not included in this report as the focus of the project was to be on low- 
achieving students. For year two, the seventh and eighth grade groups 
remained intact, to the extent possible, and the incoming sixth grade groups 
were formed using the year one strategy. For year three, the district admin- 
istration established two computer laboratories for the project, one for 
reading and one for mathematics, in response to teacher concerns that inte- 
grating computers into the classroom was disruptive and difficult to monitor; 
the administration was also responding to concerns of the local teachers' 
union. As a result of the change in design, the control group was eliminated 
and all students participated in the Jostens computer laboratories on a six- 
day, rotating schedule. Also, in each school, program students (five at a 
time) were to be on a rotating schedule to bring a computer home for a six 
week period. 
Instrumentation 

A number of evaluation instruments were developed by RBS, and approved by 
the districts, for use in the three-year study. They included the following: 

• Administrator Attitude Survey 

• Teacher Attitude Survey 

• Student Attitude Survey 

• Parent Attitude Survey 

• Computer Home Use Survey. 

The administrator and teacher attitude forms contain questions which 
address concerns regarding the new computer learning system, the ndv*» ta^es 
and disadvantages of the system, the adequacy of training and follow-up tech- 
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nical assistance, the implementation of the system, the adequacy of the compu- 
ter curriculum, and the effectiveness of the computer learning system. The 
student attitude form addresses attitude toward using the computer in school, 
and the advantages and disadvantages of the computer learning system; and the 
parent attitude form and the home use form ask about parent involvement in the 
program, their child's attitude toward participation, and the advantages and 
disadvantages of the computer learning system. The home use survey is devel- 
oped for parents of those students, who on a rotating basis take a computer 
home to increase their time on the system and their parents' involvement in 
collaborative learning. 

Achievement data were Jostens student achievement reports and standard- 
ized test data (Stanford Achievement Test (SAT) basic skills data for the 
Pittston Area School District, IOWA Test of Basic Skills (ITBS) data for the 
Hatboro-Horsham School District) . As an additional outcome measure both 
districts asked to readmini9ter the Jostens Basic Skills Inventory, designed 
to place students in the reading and mathematics curricula, to all project 
students at the end of the school year. 
Data Collection and Analysis 

Process data were collected from both school districts participating in 
the project. These data were collected primarily through frequent informal 
interviews, telephone conversations, and on-site observations. Monthly visits 
were made to each school during which time the computer coordinator was infor- 
mally interviewed along with at least one reading and one mathematics teacher 
at each grade level. The interviews focused on implementation progress and 
issues of concern. Four questions formed the basis of the informal, 
conversational interviews : 

• How has the program been running recently; have there been any 
changes? 



• Describe some parts of the program that are going well? 

• Describe some parts of the program that need to be improved? 

• Do you think the program is making a difference? 

At the Hatboro-Horsham middle school, which experienced extensive 
difficulties with the computer-based learning system during years one and two, 
these visits were scheduled, whenever possible, to coincide with the monthly 
Jostens inservice sessions. The Pittston Area School District developed an 
observation and interview schedule prior to each of the evaluator's visits. 
During both classroom and laboratory visits, students were observed working on 
computers. Also, RBS' time-on-task measure was not used to collect data, as 
observations of students working on the computer, along with teacher feedback, 
indicated that students were highly engaged while working through their 
computer assignments and there was little variability in engagement rates. 

The two school districts also submitted third year outcome data for the 
project which consisted of completed attitude surveys and student achievement 
data. Attitude surveys were to be administered at the end of the school year, 
nd year one (for the Pittston Area School District) or year two data (for the 
Hatboro-Horsham School District) were to serve as a pretest. In cases where 
pretest data were not available (e.g., for the new sixth grade students), 
surveys were administered two times, at the be^nning and end of the school 
year. 

Analyses were planned for both attitude and achievement data. Responses 
to items on the student, and parent surveys were analyzed separately for each 
evaluation instrument. Also, only the student survey data were compared to 
earlier findings; because parents did not include their names on the surveys, 
it was not possible to match their data over time. The SAT and ITBS data were 
used to examine student achievement in reading and mathematics. It should be 



noted that the 1993 ITBS data were provided to RBS during the last week in 
August, 1993 and the 1992 ITBS data were provided at the end of October, 1993, 
rather than in mid- June, as originally promised. As a result of renorming the 
test in 1992 (the previous norms were from 1988), the publisher had to develop 
a new program and restructure the scoring services which was reported to have 
taken more time than originally anticipated. The SAT was also renormed in 
1993. 

In order to assess overall achievement gains, students' 1993 standardized 
test scores (posttest) were compared to students' scores when entering the 
program (pretest). Thus, for the Pittston Area School District, the 1992-1993 
sixth grade participated one year, and the 1992-1993 seventh and eighth grades 
participated two years (the site vras only in partial operation during year 
one). For the Hatboro-Horsham School District, analyses were based on only a 
one-year participation for all 1992-1993 students, as outcome data were not 
collected during previous years due to extensive implementation problems. 
These analyses were carried out in terms of normal curve equivalents (NCEs) in 
order to look at students' achievement growth; an NCE change of zero indicates 
that students' achievement growth rate equals the rate of the representative 
national sample. In terms of the curriculum-embedded basic skills data, in 
addition to initial placement and final placement, the number of lessons 
9tudent9 completed during the year was also made available. The lesson data 
were analyzed to address project implementation and to set the stage for the 
discussion of outcomes. Both districts felt that the computer generated 
reports of student time were not valid as the software grossly underestimated 
the amount of time students spent on the computer learning system . 



Findings 

The findings presented in this section of the report relate to the eval- 
uation questions for year three. They are discussed below in two separate 
sub-sections , program implementation and program outcomes. As indicated 
earlier, because of significant changes in implementation during year three, 
the focus of this evaluation report is on the project's third and final year. 
Program Implementation 

Specific feedback on implementation of the project was obtained from 
informal interviews with teachers and computer coordinators, on-site observa- 
tions, and telephone contacts. These data are discussed below under the 
implementation evaluation question. Additional interview data is presented in 
the Program Outcomes section of the report. 

To what extent was the project implemented as planned7 

In the fall of the 1992-1993 school year, each of the two sites ordered 
and received additional hardware and software to continue their implementation 
of the project during its third year; student groups were updated/ redefined; 
parents of newly-participating students were provided with information on the 
project; and schedules were developed for student computer use and data 
collection. Following these initial preparations, the two school districts 
attempted to implement the project, as described below. 

The Hatboro-Horsham School District. During the first two years of the 
project, the Hatboro-Horsham School District experienced extensive problems 
related to the functioning of both the computer hardware and software (Beyer, 
1991; Beyer, 1992). In spite of these problems, the district continued to 
work toward the project goals. Teachers were trained in the use of the ILS, 
and they met on an on-going basis with the project coordinator, and monthly 
with the Jostens trainer, to discuss implementation issues. 
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Although some of these problems continued during the project's third 
year, the Jostens Learning System was operating more smoothly than it had 
previously. The major problem continued to be that, while working on the 
computer learning system, students were frequently presented with lessons 
which they had previously completed, which meant their progress was not being 
stored in the system. These errors were reflected in student achievement 
reports; when data on lessons completed are not stored, all other reporting 
data (e.g., average scores, times, dates) are obviously inaccurate. Addi- 
tional problems included a faulty repeater, printing difficulties with IBM 
network software, lessons freezing, printers producing pages ,of nonsense, and 
lessons being omitted from instructional units. 

To dudress problems with the operation of the Jostens Learning System, 
the computer coordinator monitored the file server and documented student 
progress on a daily basis. The documentation was then provided to Jostens 
representatives, who hypothesized several solutions, e.g., installing the 
latest update of the software (2.95); updating the CD player to be compatible 
with the file server. At one point, the documentation and data files were 
sent to a Jostens consultant in San Diego, California. After district persis- 
tence, Jostens agreed to provide an open line for support, at no additional 
cost to the district, until the technical problems were resolved. In addi- 
tion, teachers reported that the monthly staff meetings with the Jostens 
representative were helpful. This was due, in part, to the efforts of the 
computer coordinator, who set meeting agenda around the staffs* expressed 
needs. 

With the district's constant attention throughout the school year, the 
major problem of lessons repeating was reduced, but not eliminated, as was the 
level of frustration among teachers and students. Ironically, at the end of 
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the school year, the computer coordinator from the Pittston Area School Dis- 
trict solved the lesson repetition problem! In cases where IBM software is 
used, a "fairnesj switch" in the file server must be turned off. This dis- 
covery, which Jostens did not communicate, is obviously critical for all IBM 
installations . 

Early in the 1992-1993 school year, the Assistant Superintendent for 
Curriculum from the Hatboro-Horsham School District requested a meeting with 
Jostens staff to discuss the status of the Jostens project in the district. 
Attendees at the meeting, which was held in late October, included: the 
Assistant Superintendent, the middle school principal, and the project 
coordinator from the Hatboro-Horsham School District; the project coordinator 
from the Pittston Area School District; the RBS project evaluator; and six 
Jostens staff, including regional representatives and the corporation's 
Director of Testing and Evaluation. As a result of the meeting, Jostens 
recommended expanding the outcome evaluation of the project, and a number of 
alternatives were proposed, e.g., creating a criterion reference test, 
developing a writing assignment. Although these were quality suggestions, the 
district and RBS agreed that, because implementation issues were still 
paramount, the proper functioning of the computer learning system should 
continue to be the main focus, rather than an outcome evaluation. They also 
agreed that the goals of the project should not change during the project's 
third and final year. It should be noted that the recommendations offered 
were both time consuming and costly. 

During the 1992-1993 school year, the Hatboro-Horsham School District 
also attempted to implement the home learning and cooperative learning 
components of the program. In terms of the home learning component, there 
were two rotations of five computers for sixth and seventh grade students and 
one rotation of five computers for eighth grade students. Three reasons were 
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offered for why more students were not involved. First, the district decided 
that the home placement would be more meaningful if students could spend at 
least four months with the computers. Second, the computer coordinator 
experienced difficulty obtaining parental interest and support for their child 
having a computer at home. And, third, it was difficult coordinating 
home-school procedures (e.g., updating diskettes, transferring lesson results) 
and thus students were frustrated by lesson repetitions at home. In spite of 
these difficulties, the district continued to communicate with parents and to 
involve them in the project; most of these students were able to complete 
lessons and move through the curriculum. 

In the middle of the school year t after the system was up and running, 
the middle school principal met with the RBS evaluator, the computer coordina- 
tor, and the the district staff developer to plan for the implementation of 
cooperative learning in the classroom. This component of the project had been 
set back due to the malfunctioning of the system. It was decided that the 
staff developer would provide support to one teacher from each grade level 
(sixth, seventh, and eighth) in integrating cooperative learning with the 
Jostens Learning System. Cooperative learning was broadly defined as having a 
student who was using the computer on a particular day be "updated" on the 
portion of the class lesson he or she missed by a student who was not using 
the computer that day. During interviews and observations these teachers 
indicated that they were beginning to experiment with the integration of 
cooperative learning with student's use of computers. One other area with 
which teachers began experimenting was the use of Alternate Learning Pathways 
(ALPS), a component of Jostens updated curriculum. Through the use of ALPS, 
teachers could design lessons to match their current instructional topics. 
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The Pittston Area School District, The project's third year also ran 
more smoothly for the Pittston Area School District. By November, some 
technical difficulties experienced early in the year were resolved (e.g., 
malfunctioning base band unit, systems engineer taking several weeks longer 
than expected to upgrade the Jostens curriculum). Following the 
Hatboro-Horsham meeting with Jostens representatives, Pittston administrators 
scheduled a parallel meeting, which was, unfortunately, scheduled and cancpled 
on throe occasions due to inclement weather. In addition to technical 
problems, some district changes also affected the operation of the project. 
Specifically, there were three new project teachers who were, not familiar with 
the Jostens Learning System, the computer coordinator had multiple competing 
responsibilities that limited the amount of time he could devote to the 
project, and the use of the Jostens system was dramatically changed, as 
described below. 

At the beginning of this last year of the project, the Pittston Area 
School District made some major changes in how students were to be scheduled 
to use the Jostens Learning System. As previously indicated, the original 
evaluation design included an experimental or n high use" group receiving one 
hour of computer time per week and a control or "low use" group receiving 15 
minutes of computer time per week. The difficulties which the district 
experienced with this design were three-fold. First, it was difficult for 
teachers to monitor and manage students* computer time (i.e., the limited 
number of computers per class made it difficult to schedule students for 60 
minutes of computer time a week; teachers reported that students in the 15- 
minute condition frequently did not have their time recorded on the system) . 
Second, the experience of reading teachers participating in the project was 
that having students use computers during class time was disruptive, e.g., 
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students working on computers missed important instruction which meant 
teachers had to reteach portions of their lessons (some mathematics teachers 
voiced this objection as well, although to a lesser degree). And third, the 
local teacher's union was concerned about students missing regularly scheduled 
instructional time while working on the computer. 

To address these scheduling problems, the district created two Jostens 
laboratories, one for reading and one for mathematics. In November, the 
reading teachers began bringing their classes to the laboratory, once every 
six days, for an entire day, during which each class spent an entire class 
period of approximately 45 minutes working on the Jostens reading curriculum. 
Because the laboratory only had thirteen computers, and each class had more 
than 13 students, the overflow of students fro r the laboratory was assigned to 
work on computers in other reading teacher's classrooms (this meant that 
reading teachers could not use the computers in their own classrooms, as they 
were being used by these other teacher's students). 

The mathematics laboratory was not operational until mid-year, when the 
necessary wiring was completed for the thirty newly-purchased computers. 
Prior to that time, teachers rotated students on the computers in their class- 
room, in most cases aiming to provide all project students with the same 
amount of computer time each week (the weekly times estimated by teachers 
ranged from 20 to 60 minutes). The mathematics laboratory then operated on 
the same six-day schedule as the reading laboratory, but without the student 
overflow problem. 

During informal interviews, all teachers agreed that the change to a 
laboratory setting was an improvement in the operation of the Jostens Learning 
System, and many commented that the laboratory setup forced them to be more 
aware of the Jostens curriculum and more involved in their students' computer 
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work. For example, one teacher commented that t "Last year, I seldom saw what 
kids were actually doing on the computer." Another said t "I like the lab and 
monitoring kids. Some kids aren't mature enough to use the system on their 
own and need a little assistance." In addition, observations confirmed that 
teachers were comfortable in their new role of managing and monitoring an 
entire class working on computers. One teacher commented, "It's very 
structured this year and I like the control. I wasn't really happy last 
yea'." Another teacher said, "I like the lab a lot and kids like it. It's 
110 percent better." 

The Pittston Area School District also experienced some difficulty imple- 
menting the home learning portion of the project. The major problem was that 
the software discs for the home units did not function properly, and Jostens 
representatives were very slow in responding to district requests for techni- 
cal assistance. In addition, as in the Hatboro-Horsham School District, it 
was difficult to obtain parental interest in having their child have a compu- 
ter at home. After the technical problems were resolved, one rotation of 
twelve computers was used at home, from March 1993 to the end of the school 
year. In addition, cooperative learning was not part of this district's third 
year implementation because, as originally defined (i.e., pairing students 
working on and off the computer), it was incompatible with a laboratory 
setting. 

Program Outcomes 

Specific information on outcomes resulting from the two School districts' 
implementation of the computer project was gained from attitude surveys, end- 
of-year interviews, Jostens student achievement reports, and standardized 
achievement tests. The sections below describe and discuss analyses of this 
survey, interview, end achievement data. 
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Survey data . Attitude surveys were administered to students at the end 
of the school year. These data are briefly described below and are compared 
to students' initial survey results (all students were administered a student 
survey upon entering the program) . A summary of all quantifiable student 
survey data (i.e., mean ratings) is presented in Appendix A. The results of 
the parent surveys are also summarized I~alow (as noted earlier, these surveys 
were only administered in the Hatboro-Horsham School District) . 

The student survey was completed by 47 sixth, seventh, and eighth grade 
students from the Hatboro-Horsham School District. Overall, student end-of- 
year responses to the survey's 30 "yes-no" items were positive, although less 
positive than students' initial responses (i.e., 23 of the 30 items showed a 
decrease, ranging from 1 percent to 32 percent). Students indicated that the 
computer is easy to use, that they like computer work better than written 
assignments, that they can do most of the computer lessons without help from 
anyone else, that the computer helps you to correct your mistakes, and that 
your teacher knows when you make mistakes on your computer assignments (79 
percent or more of the students responded positively to these items) . Most 
students (87 percent) also recognized that it is important to do well on 
computer assignments. Fifty-four percent of the students felt that the compu- 
ter helped them to learn math better, 43 percent felt that it helped them to 
read better, and 60 percent indicated that their parents think they are 
learning from the computer. The biggest changes from students' initial 
reports were! an increase of 12 percent in students who feel that their 
teacher knows when they make mistakes on their computer assignments; and a 
decrease of 18 percent or higher in students who feel that computers make it 
fun to learn and working on the computer is fun, computers make school sub- 
jects more interesting, who like going to the computer and using the computer 



at school, who would like to go to the computer more often, and who get bored 
working on the computer by themselves. Interestingly, this latter finding is 
in direct opposition to observations and teacher reports of generally high 
student interest and engagement rates. 

It should be noted that, in spite of the decreases, the majority of 
students responded positively to most of the survey items. Also, the 
end T of-year survey appears to reflect a more negative student attitude in 
general, e.g., positive responses to the question, "Do you like school7" 
decreased from 83 to 51 percent over the pretest-posttest interval. The less 
positive end-of-year responses could be accounted for, in part, by the fact 
that the survey was administered right before summer recess. And, during 
interviews, teachers indicated that some students were "turned off" by t-hd 
tedious technical problems (i.e., repeating lessons). 

In the Pittston Area School District, end-of-year survey results from 198 
students were matched with students' initial results. Overall, the end-of- 
year responses were very positive, although in this district they were also 
less positive than students' initial responses (i.e., 23 of the 30 items 
showed a decrease, ranging from 1 percent to 44 percent). Students reported 
that they like going to the computer, they like computer work better than 
written assignments, that it is important to do well on computer assignments, 
that your teacher knows whether you make mistakes on your Computer assign- 
ments, that they can do most of the computer lessons without help from anyone 
else and that they do not have to hurry, and that the computer helps you 
correct your mistakes (80 percent or more of the students responded positively 
to these items). Fifty-seven percent of the students reported that the compu- 
ter helped them to learn math better, 41 percent felt that it helped them to 
read better, and 59 percent reported that their parents think they are 
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learning from the computer. The most positive change was an increase of nine 
percent in the same item as the Hatboro-Horsham district, i.e., students who 
felt that their teachers knows when they make mistakes on their computer 
assignments. The most negative change was, 44 percent of the respondents 
changed their mind and felt that the computer was not easy to use. Also, 
there was a decrease of 20 percent or higher in students who felt that 
computers make it fun to learn, who find the computer lessons interesting, and 
who reported that they get bored working on the computer by themselves. 
Similar to Hatboro-Horsham, this latter finding is contrary to observations 
and teacher reports. However, in spite of these decreases, the majority of 
the students in this district also responded positively to 23 of the 30 items. 

At the end of the school year, the Hatboro-Horsham School District sent 
the parent survey to all parents of students in the computer learning project. 
After repeated requests to complete and return the forms, more than half of 
the parents (56 percent) complied, representing twelve or thirteen students 
from each grade level. 

In terms of parental knowledge of and involvement in the computer 
learning system, 62 percent indicated that they were somewhat informed about 
the computer learning system at their child's school, 5 percent reported that 
they were well-informed, and 14 percent reported that they were not informed 
about the computer learning system. Most parents indicated that they were 
encouraged to visit school (mean rating of 3.4 on a 5-point scale) and that 
they were specifically invited to visit their child's classroom (63 percent). 
However, only 32 percent did visit, and only 11 percent took advantage of an 
opportunely to observe their child working on the computer. Most parents 
reported that teachers did not discuss their child's program on the computer 
(78 percent), and did not keep them informed of their child's progress on the 
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computer (mean rating of 2.5 on a 5-point scale). Interestingly, prior to 
their child's participation in the project, parents were sent a letter 
describing the project and requesting permission for their child to 
participate. During interviews, some teachers reported sending print-out9 of 
student work Home. 

In terms of parents' observation of changes in their child's attitude as 
a result of participation in the computer learning system, less than half of 
the parents reported that his or her child talks about work on the computer 
(14 percent), is more enthusiastic about school (25 percent), or likes school 
better because of computer work (19 percent). However, many parents reported 
that their child feels good about the academic progress being made on the 
computer (3.0 mean rating on a 5-point scale). Also important, many parents 
reported that their child's mathematics skills have improved because of the 
computer program (3.0 mean rating on a 5-point scale), and, to a slightly 
les9er extent, that reading skills md writing skills have improved a9 well 
(2.8 rating for each item on a 5-point scale). 

Interview dat a. During end-of-year interviews, teachers were a9ked to 
reflect on the implementation of the project and to identify its major 
strengths and weaknesses. In the Hatboro-Horsham School District, the six 
participating teachers were interviewed, along with che computer coordinator 
and 9chool principal. 

Of the eight Hatboro-Horsham staff interviewed, seven indicated positive 
attitudes toward the computer learning system and their involvement in the 
project. With one exception, all staff interviewed identified the major bene- 
fit of the Jostens system to be reinforcement of needed skills, particularly 
those below grade level skills which teachers do not have tjime to address in 
clas9. Other benefits which several staff mentioned were: computer work is 




20 



24 



an alternative means of presenting information to students who have been 
difficult to reach with traditional methods; and computer work is motivating 
to 9ome students who experience little success in class. Th ee* staff 
indicated that they are looking forward to next year, when they will have more 
freedom in how the computer learning system is being used; two staff noted 
that they personally benefited from the project as their computer literacy was 
increased and having to integrate technology caused them to rethink their 
traditional teaching strategies. 

Although most teachers felt it was too soon to comment on the impact of 
the project, three teachers did offer positive comments. First, an eighth 
grade mathematics teacher indicated that this year she was recommending more 
students for ninth grade algebra, as opposed to basic mathematics, than in 
previous years. She felt that Jostens was a contributing factor. Second, 
another teacher reported that, n A few students may have learned more from 
Jostens than they did in the classroom. ... It ' s a benefit for me with classroom 
management and for students because they don't have the needed social skills." 
A third teacher, commenting on her own personal growth, said: "I feel much 
more comfortable after three years. I can print reports and move kids around 
if I have to. I hate to give them (computers) up but I know others (teachers) 
will want them. " 

The one teacher who did not have a positive experience with the project, 
felt that, although the computer learning system has a lot of potential, she 
experienced too many implementation problems for it to be successful. For 
example, she tried to take advantage of Jostens* capability to design alterna- 
tive learning pathways (ALPs), only to find that the system did not function 
properly (i.e., lessons were included in the ALP which she did not select, and 
all selected lessons were not included). This eighth grade teacher also felt 
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that many of her students did not like using computers because of the stigma 
of being identified as part of the project. However, she did conclude that, 
"It was worth a try for three years. Although implementation didn't go well, 
we moved [in the right direction]." 

The major disadvantage cited by all other staff interviewed wa^ the large 
number of technical problem experienced, particularly during year two (e.g. , 
repeating lessons, freezing lessons, skipping lessons, inaccurate times). Two 
staff felt that integrating computers into their classroom instruction was 
difficult and that, while on the computer, students were missing classroom 
instruction on prerequisite skills for the next grade level. Several also had 
specific suggestions for Jostens around the improvement of individual lessons 
and lesson formats. 

In the Pittsto'- Area School District, the twelve participating teachers 
were interviewed, along with the computer coordinator and district administra- 
tor working with the project. When asked to identify the most positive 
aspects of the computer . learning system, all staff interviewed commented fav- 
orably on the new laboratory setting. They found the laboratory to be less 
disruptive, students did not miss classroom instruction while on the computer, 
and teachers liked being more involved in and aware of students computer work. 
For example, after students take unit tests on the computer, they are moved 
ahead in the Jostens curriculum regardless of their test score. In a computer 
laboratory setting it is much easier for a teacher to know when students are 
taking unit tests. One teacher added, "I can see what students are doing, and 
they can't use 'I was working on the computer* as an excuse." 

Similar to the Hatboro-Horsham staff, the most commonly cited benefits of 
the computer learning project by Pittston Area School District staff were: 
the reinforcement of needed skills, increased interest and motivation of lower 
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performing students (in fact, teachers indicated that many students were dis- 
appointed on days when the computer system was down and regularly scheduled 
computer time had to be canceled. Five staff also commented on the evolution 
of the project over the three-year period and the significant increase of 
student and staff computer skills. One teacher noted, "I was 'anti' three 
years ago. Now they're [the computers] fabulous; It made me a believer. I 
can actually see kids learning subtraction on the screen. " Also similar to 
Hatboro-Horsham, staff felt it was too early to determine the impact of the 
project. 

Approximately half of those interviewed commented tu the Jostens 
reporting system. With the new laboratory setting, teachers were being pro- 
vided with Jostens student achievement reports immediately following their day 
in the laboratory, and they found the reports to be timely and helpful both to 
themselves and to students as well. Three teachers also commented favorably 
on Jostens newly developed exception reports. These reports, made available 
with the latest curriculum update, helped teachers to identify the lowest 
performing and highest performing students in their classes (i.e., those 
scoring above 85 percent and below 60 percent). Two teachers and the 
administrators interviewed felt that a strength of the project was its 
integration into the regular school curriculum. For example, students were 
given grades for their work with Jostens, and these grades were included on 
students quarterly performance reports. 

Of the staff interviewed, half reported the major disadvantages of the 
computer learning system to be technical and the other half identified imple- 
mentation issues. The technical issues cited included: inconsistencies in 
some of the software (e.g., cannot see a previous screen, inconsistent or 
confusing instructions), portions of the curriculum are too elementary, the 
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fact that time was grossly underestimated. As one teacher said, "Everyone's 
really following the schedule and trying to get the minutes in, but the times 
are way off." Implementation issues included: the reading laboratory should 
have enough computers to accommodate an entire class, the project needs more 
time from the computer coordinator to respond to teacher needs and facilitate 
interaction. Sever?! teachers said their classes were mixed, i.e., n some kids 
love it, and some kids hate it." 

During interviews the administrators showed strong support for the 
project and a commitment to not only continue but also to expand the use of 
Jos tens during the next school year. The primary focus of the expansion will 
be on the use of ALPs and the integration of ALPs into outcome-base^ educa- 
tion. 

Achievement data . The first type of achievement data analyzed was the 
Jostens student achievement reports. The rationale for looking at these data 
is that, if the computer learning system is to have an impact on student 
achievement in general, it must first be demonstrated that students made 
substantial progress in the system, in terms of lessons completed. And, as 
noted earlier, both districts requested that the Basic Skills Inventory (BSI), 
the curriculum-embedded test designed for student placement in reading and 
mathematics, be readminis tered to project students at the end of the school 
year and that the data be provided to RBS for analysis. Caution should be 
used in interpreting these later analyses as the BSI was developed as a gross 
measure to place students in the Jostens curriculum, not as 2 measure of 
achievement or student progress. 

Performance records were maintained by the system for each student's 
interaction with the curriculum and the level, unit, and lesson in which the 
student was engaged. The districts were asked to provide RBS with students' 



beginning placements and number of lessons completed, in addition to end-of- 
year placements. Although, in the Jostens curriculum, the numbers of lessons 
per unit and units per grade level are riot consistent from grade to grade, the 
number of lessons completed was felt to be one of the best available measures 
of student progress in the system (as noted earlier, the computer software did 
not accurately record the amount of time students were logged on the system). 

For the Hatboro-Horsham School District, the mean number of lessons 
completed by project students, for reading and mathematics, is presented in 
Table 1, along with standard deviations. As the table indicates, sixth grade 
students completed the greatest number of lassons ov^r the course of the year 
(213), followed by eighth grade students (147). Seventh grade students 
completed relatively few lessons (56). Sixth and eighth grade students 
completed more reading than mathematics lessons, while seventh grade students 
completed few lessons in both reading and mathematics. In addition, the 
standard deviations indicate a large amount of variability within -;ach group. 

In interpreting these data it should be noted that the Jostens Student 
Achievement Report Manual recommends that students complete 12 to 15 lessons 
over an instructional interval of 4 weeks. It goes on to indicate that 
students completing fewer than 10 lessons during a 4-week period will be 
progressing too slowly to make meaningful progress. If this guideline were 
followed: the system was being implemented at aboutthe recommended level, by 
sixth grade students; at about the recommended level, by eighth grade students 
(for reading only); and at a level much lower than that' recommended by Jostens 
for achieving academic growth, by seventh grade students. 

As an aid to interpreting the lessons completed data, the total 
number of units and lessons sequenced in the various levels of the Jostens 
reading and mathematics curricula were obtained from a Jostens consultant (see 
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Table 1 



Mean Number of Reading and Mathematics Lessons Completed 
Hatboro-Horsham School District 



Grade 



N 



# 

129 
27 
100 



Lessons Completed 
Reading Mathematics 
SD 



84 
29 
47 



SD 
15 
11 
28 



Total 



27 
24 
20 



48 
15 
32 



213 
56 
147 



Table 2 below) . Although Jostens program levels are not equivalent to school 
grade levels, if students move at the recommended pace it is reasonable to 
expect that they will complete .a minimum of all lessons at one level of 
difficulty during a school year. Applying these data to those displayed in 
Table 1, it appears that the sixth grade students completed about a level and 
a third of the reading curriculum, atid a little less than a level of the 
mathematics curriculum. Seventh grade students completed about a third of a 
level of each curriculum. And eighth grade students completed a little more 
than one level of the reading curriculum, and about half of a level of the 
mathematics curriculum. 

Additional analyses were carried out on the mean BSI placement levels 
and the mean increase in program level based on the BSI pre- and posttesting. 
Included in Table 3 are the number of students in each grade (N), the mean BSI 
placement level, the mean BSI placement level based on end-of-year testing, 
and the mean pre-post increase. As the table shows, most students 
participating in the project were initially placed on the third or fourth 
grade level, with the placements for sixth grade students being closest to 
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their actual grade levels ♦ In terms of the pr*-post increases, mean gainsfor 
seventh and eighth grade mathematics were approximately one and one half 
years; the mean gains for sixth grade mathematics and sixth, seventh, and 
eighth grade reading were between one half and one year. The inconsistencies 
for the seventh grade, between the BSI and lessons completed analyses should 
be noted. In interpreting these gains it should also be noted that placement 
points for the reading curriculum, Levels 2 through 6, are the first lesson of 
each level (Level 1 has 3 placement points); placement points for the 
mathematics curriculum are at the first lesson or mid-point of each level. 



Table 2 

Total Number of Units and Lessens in the Jostens Curricula* 



.Curriculum Area/ 
Level 


Number of Units 


Number of Lessons 


Mathematics 






Level 1 


11 


100 


Level 2 


13 


111 


Level 3 


11 


91 


Level + 


' ' 11 


77 


Level 5 


12 


80 


Level 6 


12. 


71 


Reading 






Level 1 


8 


91 


Level 2 


10 


109 


Level 3 


10 


99 


Level 4 


10 


93 


Level 5 


12 


106 


Level 6 


11 


101 



*The numbers in the table were obtained via telephone from a Jostens 
consultant, August- September, 1993 . 



Thus, the Jo9tens BSI was developed as a placement tool, not a measure of 
progress, and therefore is most likely not a sensitive indicator for measuring 
student achievement , 



Table 3 

Student Basic Skills Inventory (BSI) Scores by Grade 
Hatboro-Horsham School District 



Mean Placement 



Grade/Subject 


N 


Initial 


End-of-Year 


Decrease 


6th Grade 












Reading 


25 


4 


.4 


5.0 


.6 


Mathematics 


25 


3 


.6 


4.3 


.7 


7th Grade 












Reading 


24 


3 


3 


4.1 


.8 


Mathematics 


24 


2 


5 


3.9 


1.4 


8th Grade 












Reading 


18 


4 


4 


5.0 


.6 


Mathematics 


18 


4 


2 


5.7 


1.5 
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Tables 4 and 5 present similar analyses of the Jostens student report 
datf» for the Pittston Area School District. As indicated in Table 4, the 
sixth grade implemented the program at a higher level than the seventh grade 
for mathematics, and at a slightly lower level than the seventh grade for 
reading (unfortunately, data for grade 8 were not provided to RBS) . However, 
using both Jostens' guideline for the number of lessons to be completed each 
week and also the total number of lessons in each curricula level (see Table 
2) to interpret the data, both grade levels were progressing through the 
curricula at a rate lower than that recommended by Jostens for achieving 
academic growth. 
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Table 4 



Mean Number of Reading and Mathematics Lessons Completed 
Pittston Area School District 



Grade N 



6 
7 

8* 



44 
56 



Lessons Completed 
Reading Mathematics 
SD # 



41 
55 



11 
17 



79 
29 



SD 



29 
19 



Total 



118 
84 



*Data not provided to RBS. 



Table 5 

Student Basic Skills Inventory (BSI) Scores by Grade 
Pittston Area School District 



Grade/Subject 



Mean Placement 



Initial 



End-of-Year 



Increase/ 
Decrease 



6th Grade 

Reading 
Mathematics 

7th Grade 



70 
70 



4.5 
3.9 



4.1 
3.9 



-.4 

0 



Reading 
Mathematics 



74 
74 



4.0 
4.0 



8th Grade 



Reading 
Mathematics 



67 
67 



3.8 

3.9 



4.7 
4.6 



.9 
. 7 



*Data not provided to RBS. 
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Analyses of the mean BSI placement levels and the mean increase in 
program level based on the BSI pre and posttesting are presented in Table 5. 
Similar to the Hatboro-Horsham district, most project students were initially 
placed on the third or fourth grade level (unfortunately, initial placement 
data for the seventh grade were not provided to RBS) . In terms of pre-post 
increases, the sixth grade had a small decrease in reading, and no change in 
mean mathematics placement level, and the mean gains for eighth grade reading 
and mathematics were close to one grade level (.9 and .7, respectively). 

In addition to student progress on the Jostens curriculum, another focus 
of the evaluation is on the extent to which progress on the computer system is 
transferable to standardized achievement tests. Tables 6 and 7 present analy- 
ses of ITBS mathematics and reading scores for the Hatboro-Horsham School 
District. The scores are presented as normal curve equivalents (NCEs) by 
grade and group, for tests administered in the spring of 1992 (pretest) and 
1993 (posttest) . Students' scores in both the experimental and control groups 
are matched for the two years. 

NCEs are normalized standard scores, with 50 indicating the national 
average or grade level. An NCE gain of zero from one year to the next would 
signify that students maintained the same relative standing with respect to 
the norm group, which is what would be expected to occur without any special 
program or intervention. However, the general rule of thumb is that only NCE 
changes of three or more are considered to be educationally significant. As 
the comparisons in Tables 6 and 7 indicate, overall, students in the Jostens 
program had more significant increases than the control students. Four out of 
six comparisons were significant for the experimental group (i.e., seventh and 
eighth grade reading and mathematics), whereas only two out of six comparisons 
were significant for the control students (i.e., eighth grade reading and 



Table 6 





Mathematics Achievement Test Scores (ITBS) 
Experimental and Control Groups 
Hatboro-Horsham School District 


for 






Grade/ 




ITBS (Mean 


NCE) 


NCE 






Group 


N 


1992 


1993 


Gain 


t 




6th Grade 

Experimental 
Control 


25 
22 


40 
47 


41 
48 


+1 
+1 


ns 
ns 




7th Grade 

Experimental 
Control 


24 
22 


33 
43 


40 
46 


+7 
+3 


-3 
ns 


09* 


8th Grade 

Experimental 
Control 


19 
18 


43 
43 


49 
49 


+6 
+6 


-2. 
-3. 


07* 
02* 


* p < .05 




Table 7 












Reading Achievement Test Scores (ITBS) for 
Experimental and Control Groups 
Hatboro-Horsham School District 






Grade/ 




ITBS (Mean 


NCE) 


NCE 






Group 


N 


1992 


1993 


Gain 


t 




6t.h Grade 

Experimental 
Control 


25 
22 


42 
50 


43 
49 


+1 
-1 


ns 
ns 




7th Grade 

Experimental 
Control 


24 
22 


39 
44 


46 
48 


+ 7 
+ 4 


-3. 
ns 


18* 


8th Grade 
Experimental 
Control 


19 
18 


46 
44 


52 
50 


+ 7 
+6 


-2. 
-3. 


96* 
50* 



* p < .05 
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mathematics). Also, all six comparisons showed a positive increase for the 
experimental group, and five out of six were positive for the control group. 
Considering this was only a one year implementation of the computer learning 
system, these data seemed promising. Further analyses were then conducted. to 
compare experimental and control groups at each grade level, using Analysis of 
Covariance to statistically control for any initial group differences on the 
pretest. The results of these analyses showed no significant differences 
between posttest means at any of the three grade levels, for reading or 
mathematics (summaries of these analyses are presented in Appendix B). 

Tables 8 and 9 present results of analyses of the standardized test data 
(SAT) for the Pittston Area School District. Three years of data are included 
for this district, as compared to two years for Hatboro-Horsham, because this 
district implemented the project during the 1991-1992 school year (1991-1992 
data were included in the previous evaluation report, Beyer, 1992). Also 
included in the tables are mean NCE scores for all non-project students (whose 
scores could be matched) in the three grade levels. These data from non- 
project students are included for the purpose of identifying district trends 
in test scores, not for comparing the actual scores of the two groups. 

As Tables 8 and 9 show, the experimental or project group did not make 
anv significant gains (comparisons could not be made for grade 6, as data for 
this group in the 1991-1992 fifth grade were not available). Upon examining 
the grade comparisons for these students, sev nth grade mathematics and 
reading scores showed a statistically significant decrement from the expected 
level of achievement, as did eighth grade mathematics scores. The other two 
comparisons were non-significant changes of 0 and -2 NCEs . Interestingly, the 
pattern for the not at-risk group was similar, with seventh grade mathematics 
and reading scores and eighth grade mathematics scores also showing a 
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Table 8 



Mathematics Achievement Test Scores (SAT) for 
Experimental and Not at Risk Groups 
Pittston Area School District 



Grade/ SAT (Mean NCE) NCE Gain t 

Group N 1991 1992 1993 91-2/92-3 



6th Grade 

Experimental 58 
Not At Risk** 158 



47 
61 



.-/„ 
.-/„ 



7th Grade 

Experimental 58 
Not At Risk** 120 



58 
71 



50 
61 



--/- 8 
~/-10 



■-/ 7.03* 
■-/10.81* 



8th Grade 

Experimental 84 
Not At Risk** 82 



54 
75 



46 
65 



44 
65 



-8/-2 8.80*/ 1.94 

■10/0 12.31*/ -.55 



* p < .05 

** This group includes all non-project students and is included in the table to 
identify district trends. 



Table 9 

Reading Achievement Test Scores (SAT) for 
Experimental and Not at Risk Groups 
Pittston Area School District 



Grade/ \ SAT (Mean NCE) NCE Gain t 

Group N 1991 1992 1993 91-2/92-3 



6th Grade 

Experimental 58 
Not at Risk** 158 



47 
57 



-/- 



7th Grade 

Experimental 58 
Not at Risk** 120 



51 
61 



45 
57 



.-/-6 
--/-4 



--/ 3.53* 
--/ 3.79* 



8th Grade 

Experimental 84 
Not at Risk** 82 



47 
68 



47 
62 



47 

66 



0/0 -.14 / .13 

•6/+4 4.72*/-3.05* 



* p < .05 

** This group includes all non^-project students and is included in the table to 
identify district trends. 

O 33 

ERLC 37 



statistically significant decrement from the expected level of achievement. 
The one additional finding for this non-project group was for the eighth 
grade, with a statistically significant decrease in r. ,ading achievement 
scores, from 1991-1992 and a statistically significant increase in reading 
achievement scores, from 1992-1993 . 

In summary, there is no evidence in the Pittston Area School District of 
accelerated achievement levels on the SAT resulting from the computer learning 
system program. However, a number of factors must be considered in 
interpreting these data: first, the SAT test administered at the end of the 
1993 school year was based on a new set of revised norms which were different 
from those used in previous years; second, the trends evident in the 
standardized test scores of project students are almost identical those of the 
school's higher-achieving, non-project students; third, a control group was 
not available for statistical comparison; and also significant is the fact 
that the quality of the implementation was not at a high level throughout the 
life of the project. 

Conclusions and Implications 
Presented below are general conclusions which relate to the project's 
four evaluation questions, followed by some implications of the evaluation's 
findings and conclusions. 
Conclusions 

To what extent was the project implemented as planned ? During years one 
and two of the CAI/Cooperative Learning Project, the two participating school 
districts experienced many difficulties with the Jostens Learning 
Corporation's computer-based learning system. These problems involved both 
the computer software and hardware, and implementation issues (i.e., how to 
best integrate the project and its components into the regular school 
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curriculum). During year three, the computer learning system was more fully 
operational at both sites; many problems were resolved, while others were less 
disruptive. To some extent, the Hatboro-Horsham site continued to experience 
hardware/software/networking problems, and the Pittston site addressed 
implementation difficulties which necessitated changes in the project design 
(unfortunately the changes resulted in the loss of a comparable control group, 
which is needed to minimize threats to internal and external validity) . 
However, to their credit, both school districts continued to focus on project 
goals over the three-year period and to deal with and overcome many technical 
and practical barriers. As a result of their efforts, the Hatboro-Horsham 
district was thus able to reach a fairly high level of implementation and to 
introduce the various components of the project, as appropriate; the Pittston 
district's level of implementation of its revised design was more moderate. 

To what extent does the project enhance mathematics and language arts 
achievement for the participating students as measured by an increase in 
standardized test scores ? In terms of student achievement , the first data 
examined were student progress on the Jostens curriculum. Specifically, the 
variables of interest were number of lessons completed, and pre and posttest 
scores for the Jostens Basic Skills Inventory (BSI) . For the Hatboro-Horsham 
School District, the lessons completed data showed high implementation levels 
for sixth and eighth grades. Although the BSI results showed sixth grade to 
have the lowest gains, it is difficult to interpret these findings as this 
curriculum-embedded test is not technically valid for measuring student 
progress. This pattern of results for the analysis of completed Jostens 
lessons and BSI data was similar for the Pittston Area School District, 
alttn m,h the total number of lessons completed for the middle school grades in 
thii 'i'.strict was generally lower. 
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Analysis of standardized test scores in the Hatboro-Horsham middle school 
(ITBS) provided evidence that all Jostens students had NCE gains in both 
reading and mathematics, and for seventh ani eighth grade these gains in 
reading and mathematics were statistically significant. In comparison, only 
eighth grade gains were significant for the control group. These results are 
quite impressive considering the technical difficulties experien >d by this 
district during years one and two. However, grade-level comparisons between 
experimental and control groups failed to reach significance using Analysis of 
Covariance. It should be noted that, time spent on Jostens, which in most 
cases focused on reinforcing basic skills, did not negatively impact student*? 
performance. Analyses of Stanford Achievement Test (SAT) scores for the 
Pittston middle school did not provide evidence of score improvement. 
However, the district's elimination of the control group makes it difficult to 
d::aw conclusions about project impact. And the fact that the district trend 
has been for SAT scores to decline, questions the assumption that, without any 
new programs or Special treatments, SAT scores would remain stable relative to 
the norm group over the pre to posttest interval. 

To what extent does the project enhance positive student and parent atti - 
tudes toward learning, the integrated learning system, and education in 
general? Overall, students had positive attitudes toward the computer- 
assisted instructional program* although unlike teachers and administrators, 
student enthusiasm decreased dver time. This could be interpreted as atti- 
tudes becoming more realistic with experience. That is, staff anxiety 
decreases as they grow to develop an appreciation for computer-assisted 
instruction and a more positive attitude, and students' initial excitement 
about the nev technology wanes as they become accustomed to the "newness" of 
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the innovation and understand it is a serious component of their school 
curriculum. 

Information on parent attitude was obtained from parents in the Hatboro- 
Horsham School District at the end of the school year (it was not feasible to 
administer the survey on two occasions). Although overall, parents reported 
being informed about the project, they also indicated being minimally involved 
in their child's computer work i.e., through discussions with their child, 
through information from teachers, or through observations. Ironically, the 
Pittston Area School District, which did not administer the parent survey, was 
incorporating students' performance on into quarterly progress reports and 
teacher /parent conferences. 

Hatboro-Horsham parents who responded to the survey felt that the impact 
of the computer learning system on their child was more in terms of improving 
specific skills (e.g., mathematics skills) rather than improving their child's 
attitude toward school. Also important, most of these parents reported that 
their child feels good about academic progress being made on the computer. 

To what extent does the project enhance positive teacher and administra - 
tor attitudes toward the integrated learning ^ystem and cooperative learning ? 
During end-of-year interviews, teachers and administrators had very positive 
attitudes toward the Jostens computer learning system. Hatboro-Horsham staff 
had worked through two years of extensive technical problems and were finally 
seeing the benefits of the system and its capabilities during year three. 
Pittston staff were pleased that their implementation concerns were addressed 
and felt that the laboratory setup was a major improvement in how the project 
was being implemented. They no longer had to deal with classroom management 
issues or satisfy experimental time constraints. In addition, it is clear 
that, in both districts, initial staff anxiety about the program and the 
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operation of the new technology decreased with experience, over time. In 



general, teachers came to realize that lower-achieving students can benefit 
from working on computers, at their own rate, and in non- threatening 
situations . 
Implications 

The general implications of the findings and conclusions are summarized 
as follows: 

• For a project to succeed, what is originally proposed must be not only 
well conceived, but also realistic. The implementation of an innova- 
tion which includes technology is a major challenge. An effective 
strategy to include additional components (e.g. , collaborative 
learning at home and in the classroom) might be to phase them in, in a 
step-wise fashion, during successive years, which is what actually 
occurred in one of the two implementing districts (the Hatboro-Horsham 
School District ) . 

• It is also critical, in planning, to define each project component and 
how it is to be implemented and monitored. For example, cooperative 
learning, although a "hot" educational topic, has a strong research 
base (e.g., Slavin, 1989) which must be taken into account when 
proposing a "cooperative learning project." That is, we know that 
successful methods are those that include instruction, structure, 
individual accountability, group goals and rewards, and long-term 
teacher support . ■ 

• Planning for a project such as this one, that involves federal funds, 
must take into account the realities of the funding cycle. For exam- 
ple, in this project the timeline was as follows: requests to pur- 
chase new equipment were developed in the Spring of each year, 
approval was communicated in late August, yet, the federal government 
did not release funds until October 1, at which time a purchase order 
had to be written by the district and processed. As a result, newly 
purchased equipment did not get up and running until the second half 
of each school year, and implementation was obviously delayed. 

• Effective assistance, from inside or outside the school, i.s strongly 
associated with good project outconu s (Huberman & Miles, 1984). When 
technology is included, this support should include initial training 
by the vendor (for both the project coordinator and teachers), ongoing 
meetings to discuss technical and implementation issues (i.e., on the 
job training), on-going communication with colleagues who are also 
implementing the project, and the availability and accessibility of a 
project coordinator who works closely with the staff and is sei.oitive 
to their needs . 
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• Interestingly, each of the sites involved in the project had a differ- 
ent response to the innovative use of computers. One site attempted 
to implement on a school-wide basis, and the project coordinator had 
multiple competing responsibilities.. This site ended up creating 
computer laboratories, and thus resisted the kind of instructional 
innovation originally proposed by the project. The second site, which 
appeared to have more positive outcomes as a result of its implementa- 
tion, overcame much initial staff resistance to change and integrated 
the computers within the classroom. This site had fewer students 
involved and had a full-time project coordinator. However, some 
suggest that, what is important is "letting computers in the door." 
That is, as teachers become more familiar with computer technology, 
needed changes in curriculum and instruction will begin to occur and 
schools will become more compatible with society (e.g., Collins, 
1991). 

This evaluation report recognizes the extraordinary efforts on the part 
of the two school districts in attempting to implement the CAI/Cooperative 
Learning Project. Although computer-assisted instructional programs such as 
these commonly experience setbacks and delays, the amount of difficulty 
experienced by these two districts was exceptional. It is hoped that the 
report will be useful to the districts as their integration of educational 
technology continues to grow and evolve. Clearly, technological resources 
such as computers are essential for preparing students for the 21st century. 
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APPENDIX A 

Mean Student and Parent Survey Responses for Rated Items 



4 ij 

FRir 



Student Survey 
Hatboro-Horsham Middle School (N=47) 



Percent Pre-Post 
Responding Difference 
(Post) 



1. 


Item 

Do you like school? 


Yes 
51 


No 
49 


-32 


2. 


Is the computer easy to use? 


98 


2 


+ 4 


3. 


Is working on the computer fun? 


48 


52 


-20 


4. 


Do computers make it fun to learn? 


53 


47 


-26 


5 . 


Do you learn a lot on the computer? 


52 


48 


-18 


6. 


Do computers make school subjects 
more interesting? 


34 


66 


-19 


7. 


Do you get bored working on the 
computer by yourself? 


76 


24 


+21 


8 . 


Do you need much help when working 
on the computer? 


7 


93 


+ 3 


9. 


Does your computer give you help 
when you need it? 


67 


33 


-12 


10. 


Does the computer help you correct 
your mistakes? 


79 


21 


-4 


11. 


Do you have to hurry when you work 
on the computer? 


9 


91 


-4 


12. 


Do you like computer work better than 
written assignments? 


83 


17 


0 


13. 


Is it important to do well on your 
computer assignments? 


87 


13 


-4 


14. 


Does working on the computer help 
you do better in school? 


44 


56 


-8 


15. 


Does your teacher know whether you make 
mistakes on your computer assignments? 


79 


21 


+12 


16. 


Do you get good grades when you work 
hard in school? 


89 


11 


-7 


17. 


Do you like going to the computer? 


48 


52 


-20 
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18. 


Would you like to go to the computer 
more often? 


30 


70 


-33 


19. 


Have you worked on <? computer in school 
before this year? 


89 


11 


-2 


20. 


Do you have a computer at home? 


43 


57 


+8 


21. 


Do you like using the computer at school? 


59 


41 


-19 


22. 


Has the computer helped you to learn 
math better? 


54 


46 


-7 


23 . 


Has the computer helped you to read better? 


43 


57 


-6 


24. 


Has the computer helped you to write better? 


23 


77 


+1 


25. 


Has the computer helped you to understand 
science better? 


9 


91 


-13 


26. 


Can you do most of the computer lessons 
without help from anyone else? 


89 


11 


-11 


27. 


Are the computer lessons interesting? 


52 


48 


-7 


28. 


Do your computer lessons help you do 
work in the classroom better? 


38 


62 


-15 


29. 


Is your time with the computer the best 
part of your day? 


13 


87 


-11 


30. 


Do your parents think you are learning 


60 


40 


-10 



from the computer? 



Student Survey 
Pittston Area Middle School (N-198) 



Percent Pre-Post 
Responding Difference 
(Post) 



1. 


Item 

Do you like school? 


Yes 
81 


No 
19 


+12 


2. 


Is the computer easy to use? 


52 


48 


-44 


3 . 


Is working on the computer fun? 


67 


33 


-17 


4. 


Do computers make it fun to learn? 


64 


36 


-20 


5. 


Do you learn a lot on the computer? 


57 


43 


-15 


6. 


Do computers make school subjects 
more interesting? 


57 


43 


-14 


7. 


Do you get bored working on the 
computer by yourself? 


61 


39 


+26 


8. 


Do you need much help when working 
on the computer? 


8 


92 


+4 


9. 


Does your computer give you help 
when you need it? 


70 


30 


-10 


10. 


Does the computer help you correct 
your mistakes? 


81 


19 


-8 


11. 


Do you have to hurry when you work 
on the computer? 


20 


80 


-1 


12. 


Do you like computer work better 
than written assignments? 


88 


12 


-3 


13 . 


Is it important to do well on your 
computer assignments? 


90 


10 


-3 


14. 


Does working on the computer help 
you do better in school? 


56 


44 


-13 


15. 


Does your teacher know whether you make . 
mistakes on your computer assignments? 


82 


18 


+9 


16. 


Do you get good grades when you work 
hard in school? 


90 


10 


-6 


17 . 


Do you like going to the computer? 


80 


20 


-6 




18. Would you like to go to the computer 
more often? 



73 



27 



-9 



19. Have you worked on a computer in 
school before this year? 

20. Do you have a computer at home? 

21. Do you like using the computer 
at school? 



94 6 

36 64 
76 24 



-1 

+5 
-11 



22. Has the computer helped you to 
learn math better? 



57 



-11 



23. Has the computer helped you to 
read better? 



41 



59 



24. Has the computer helped you to 
write better? 

25. Has the computer helped you to 
understand science better? 



15 



13 



85 



87 



-3 



26. Can you do most of the computer lessons 
without help from anyone else? 

27. Are the computer lessons interesting? 

28. Do your computer lessons help you do 
work in the classroom better? 



95 

55 
46 



45 
54 



+ 1 

-24 
-18 



29. Is your time with the computer the 
best part of your day? 

30. Do your parents think you are learning 
from the computer? 



35 



59 



65 



41 



-8 



-17 




APPENDIX B 

Summary Tables of Analysis of Covariance for Hatboro-Horsham 

Standardized Test Data 
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Analysis of Covariance: Comparison of Experimental and Control Groups 
on Reading Achievement Test Scores (ITBS) 
(Hatboro-Horsham School District) 



Grade/ 
Group 



1993 ITBS (Mean NCE) 



Obtained 



Adjusted 



Grade 6 

Experimental 
Control 



25 
22 



43 
49 



44 
47 



,63 (ns) 



Grade 7 

Experimental 
Control 



24 
22 



46 
48 



48 
47 



,08 (ns) 



Grade 8 

Experimental 
Control 



19 
18 



52 
50 



51 
50 



10 (ns) 



Analysis of Covariance: Comparison of Experimental and Control Groups 
on Mathematics Achievement Test Scores (ITBS) 
(Hatboro-Horsham School District) 



Grade/ 
Group 



N 



1993 ITBS (Mean NCE) 
Obtained Adjusted 



Grade 6 

Experimental 
Control 



25 
22 



41 
48 



42 
47 



2.46 (ns) 



Grade 7 

Experimental 
Control 



24 
22 



40 
46 



43 
43 



,10 (ns) 



Grade 8 

Experimental 
Control 



19 
18 



49 
49 



49 
48 



,05 (ns) 
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